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Abstract: While Elizabeth Barrett Browning counted 25 ways in which she loves her 
husband in her poem, “How Do I Love Thee? Let me Count the Ways” ive identified only 
eight ways to evaluate the potentialfor success ofa federal research grant proposal. This may 
be surprising, as it seems upon initial glance of the review criteria used by various federal 
funding agencies that each has its own distinct set of “rules” regarding the review of grant 
proposals for research and scholarship. Much of the grantsmanship process is dependent upon 
the review criteria, which represent the funders’ desired impact of the research. But since 
mostfunders that offer research grants share the overarching goals of supporting research that 
(1) fits within its mission and (2) will bring a strong return on its financial investment, 
the review criteria used to evaluate research grant proposals are based on a similar set of 
fundamental questions. In this article, we compare the review criteria of 10 US federal 
agencies that support research through grant programs, and demonstrate that there are 
actually only a small and finite number of ways that a grant proposal can be evaluated. 
Though each funding agency may use slightly different wording, wefound that the majority 
of the agencies’ criteria address eight key questions. Within the highly competitive landscape 
of research grantfunding, new researchers mustfind supportfor their research agendas and 
established investigators and research development offices must consider ways to diversify their 
funding portfolios, yet all may be discouraged by the apparent myriad ofdifferences in review 
criteria used by various funding agencies. Guided by research administrators and research 
development professionals, recognizing that grant proposal review criteria are similar across 
funding agencies may help lower the barrier to applying forfederalfunding for new and early 
career researchers, or facilitate funding portfolio diversification for experienced researchers. 
Grantmakers are furthermore provided valuable guidance to develop and refine their own 
proposal review criteria. 

Keywords: Funding portfolio, research grants, proposal revieiv criteria, peer revieiv, federalfunding, 
grantsmanship, proposal development, research development, research administration 
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Introduction 

The research funding landscape in the United States is highly competitive, with flat or shrinking 
budgets for investigator-initiated research programs at most federal agencies (American 
Association for the Advancement of Science (AAAS), 2014). Taking biomedical research as 
an example, in 2014, the National Institutes of Health (NIH) budgeted $15 billion to fund 
research project grants, an amount that has essentially remained the same since 2003 (AAAS, 
2014; Federation of American Societies for Experimental Biology, 2014). At the same time, the 
number of research grant applications has steadily increased, from close to 35,000 in 2003 to 
51,000 in 2014. The result has been a stunning 30% drop in funding success rates, from 30.2% in 
2003 to 18.8% in 2014. Other federal agencies that fund research, including the National Science 
Foundation (NSF), Office of Veterans Affairs (VA), and Department of Defense (DoD), are 
feeling the similar sting of budget restrictions. 

Within this tenuous funding environment, it has become essential that investigators and research 
development offices sustain their research programs by continuing to encourage new researchers to 
apply for grant support and encouraging established researchers to diversify their funding portfolios. 
New researchers benefit from clear information about the federal grant process, and experienced 
researchers benefit from considering funding opportunities from federal funding agencies, national 
organizations and advocacy groups, state agencies, private philanthropic organizations, regional or 
local special interest groups, corporations, and internal institutional grant competitions that may 
not be their typical targets for support. With increasing competition for grant funding, investigators 
who might be accustomed to one set of rules for preparing grant proposals may become quickly 
overwhelmed by the prospect of learning entirely new sets of rules for different funding agencies. 

Tet this process is not as daunting if we start from the perspective that any funder that offers research 
grants has essentially the same goal: to support research that fits within its mission and will bring 
a strong return on its financial investment (Russell & Morrison, 2015). The review criteria used 
to evaluate research grant proposals reflect the funders approach to identifying the most relevant 
and impactful research to support (Geever, 2012; Gerin & Kapelewski, 2010; Kiritz, 2007). Thus, 
planning and preparing a successful grant proposal depends on a clear understanding of the review 
criteria that will be used. These criteria directly inform how the proposal content should be presented 
and how much space should be afforded to each section of the proposal, as well as which keywords 
should be highlighted. It may seem that each funder—federal, state, local, private—has its own 
distinct set of rules regarding the preparation and review of grant proposals, and that each funder 
uses specific jargon in its review process. However, because all funders aim to support research that 
is relevant and impactful, we suggest that the mandatory review criteria used to evaluate research 
grant proposals are based on a set of fundamental questions, such as: Does this research fit within 
the funder’s mission? Will the results of this research fill a gap in knowledge or meet an unmet need? 
Do the investigators have the skills and resources necessary to carry out the research? 

In this article, we examine the research grant proposal review criteria used by 10 US federal 
agencies to demonstrate that there exist only a small and finite number of ways that federal research 
grant proposals are actually evaluated. Our goal is to help research administrators and research 
development professionals empower investigators to more confidently navigate funder review 
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criteria, thereby lowering the barrier to first-time applicants or to grant portfolio diversification 
for more established researchers. Recognizing that research proposal review criteria are aligned 
across federal funding agencies can also help proposal writers who might be faced with other 
funding opportunities in which the review criteria are not clearly defined. On the flip side of that 
equation, understanding that review criteria are based on the same core goals can help grantmakers 
as they develop and refine review criteria for their funding opportunities. 

Observations 

We performed an online search of 10 US federal agencies’ (NIH, NSF, VA, Department of 
Education [ED], DoD, National Aeronautics and Space Administration [NASA], Department 
of Energy [DOE], United States Department of Agriculture [USDA], National Endowment 
for the Humanities [NEH], and National Endowment for the Arts [NEA]) websites to identify 
policies and procedures related to their research grant proposal review process. The NIH Office of 
Extramural research (OER) website provided the greatest detail and transparency with regard to the 
review criteria and review process used for evaluating research grant proposals (National Institutes 
of Health, 2008a; 2008b; 2015a), and served as a starting point for our analysis of the review criteria 
for the other nine agencies. We developed key questions corresponding to each of the NIH review 
criteria, and then aligned the review criteria of the remaining nine agencies with these key questions. 

Federal grant program guidance and policy changes occur frequently; the links to online resources 
for research grant proposal policies for each of the various funding agencies included in our analysis 
were current as of August 10,2015. Note that our analysis includes information from the National 
Institute on Disability and Rehabilitation Research (NIDRR) program as administered by ED. On 
June 1,2015, the NIDRR was transferred from ED to the Administration for Community Living 
(ACL) in the US Department of Health and Human Services (DHHS), and is now called the 
National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) 
Field-Initiated Program. Our analysis ofNIDRR was current as ofMay 4, 2015. 

Also note that there is variability between different research grant programs within each federal 
agency. We included in our analysis review criteria from the DoD Congressionally Directed 
Medical Research Programs (CDMRP), the USDA National Institute of Food and Agriculture, 
the NEH Digital Humanities Start-up program, and the NEA ART WORKS program. Criteria 
for NASA research programs were compiled from numerous NASA Research Announcements. 

The NIH review criteria 

The NIH criteria emphasize clinical, interdisciplinary, and translational biomedical research 
(National Institutes of Health, 2008a). Reviewers are instructed to evaluate research grant 
proposals based on how well five core review criteria are met: Significance, Innovation, Approach, 
Investigator(s), and Environment (Table 1) (National Institutes of Health, 2015a; 2015b). 
Assigned reviewers consider each of the five core review criteria and assign a separate score for 
each using a 9-point scale. These ratings are included in a summary statement that is provided to 
the researcher, whether or not the entire study section ultimately discusses the proposal. 
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Each of the five core review criteria can be simplified into a general question. The Significance 
criterion asks reviewers to consider “Why does the research matter?” Reviewers look for whether 
the proposed project will address an important problem or critical barrier to progress in the field, 
and whether the knowledge gained from the proposed research will advance scientific knowledge, 
technical capacity, or clinical practice to drive the field forward. Innovation translates into 
“How is the research new?” Reviewers consider how the proposed research challenges current 
thinking with novel concepts, approaches, tools, or treatments. Approach asks, “How will the 
research be done?” Reviewers assess the proposed research strategy, methodology, and analyses 
and determine whether they are appropriate to achieve the aims of the project, and how riskier 
aspects of the proposal might be handled with alternative approaches. The remaining two core 
criteria evaluate the context in which the research will be done—defined as the collective set 
of resources, equipment, institutional support, and facilities available (Environment)—and what 
is special about the people doing the research (Investigator). For the Environment criterion, 
reviewers evaluate whether the resources and institutional support available to the investigators 
are sufficient to ensure successful completion of the research aims, including any unique features 
such as access to specific subject populations or collaborative arrangements. For the Investigator 
criterion, reviewers determine whether the primary investigator (PI), other researchers, and any 
collaborators have the experience and training needed to complete the proposed research, as well 
as how collaborators will combine their skills and work together. 


Table 1. The NIH core review criteria for research project grant proposals 3 


Review Criterion 

Key Question 

Significance 

Why does the research matter? 

Innovation 

How is the research new? 

Approach 

How will the research be done? 

Environment 

In what context will the research be done (e.g., facilities, resources, 
equipment, and institutional support)? 

Investigator 

What is special about the people doing the research? 

Overall Impact b 

What is the return on investment? 


a See National Institutes of Health, 2015a; 2015b. 

b While Overall Impact is not considered as a core review criterion, it asks reviewers to take into 
consideration the five core review criteria as they assess the likelihood of the project to have a 
strong and sustained influence on the research field. 

NIH, National Institutes of Health. 


The five core review criteria ratings, in addition to other proposal-specific criteria, are then used 
to determine an Overall Impact/Priority Score (National Institutes of Health, 2015a; 2015b). 
This score reflects the reviewers’ assessment of the “likelihood for the project to exert a sustained, 
powerful influence on the research field(s) involved.” An application does not need to have 
exemplary scores in all criteria in order to be judged as likely to have a high overall impact. For 
example, a project that by its nature is not highly innovative may nevertheless be deemed essential 
to advance knowledge within a field. A 2011 study by the National Institutes of General Medicine 
Science (NIGMS) examined the correlation between the core review criteria scores and the 
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Overall Impact score and found that reviewers weighted certain criteria more heavily than others, 
in the following order: Approach > Significance > Innovation > Investigator > Environment 
(Rockey, 2011). Thus, the quality of ideas appeared to matter more than investigator reputation, 
a particularly good finding for new investigators (Berg, 2010a; 2010b; 2010c). These findings of 
relative importance of the core review criteria by reviewers also suggest that, in terms of space, it 
makes sense for proposers to utilize more pages of the proposal narrative to address aspects of their 
approach and the research project’s significance than on the environment supporting the project. 

Other agencies have formalized systems for weighting grant proposal review criteria. For 
example, the ED NIDRR standard selection criteria are weighted using a points designation 
(US Department of Education, 2014): Design of Research Activities (50 pts); Importance of 
the Problem (15 pts); Project Staff (15 pts); Plan of Evaluation (10 pts); and Adequacy and 
Accessibility of Resources (10 pts). Similar to NIH reviewers, ED weights research design and the 
importance of the problem more heavily than staff or resources when evaluating grant proposals 
(Committee on the External Evaluation of NIDRR and Its Grantees, National Research Council, 
Rivard, O’Connell, & Wegman, 2011). 

How do the NIH review criteria compare to those of other federal agencies ? 

The most straightforward comparison of research grant review criteria is between the NIH and 
NSF, which together make up 25% of the research and development budget in the US (AAAS, 
2014). The NSF criteria emphasize transformative and interdisciplinary research (National 
Science Foundation, 2007), and involve three (3 ) guiding principles, two (2) review criteria , and 
five (5) review elements (National Science Foundation, 2014). The two review criteria used by the 
NSF are Intellectual Merit, which encompasses the potential to advance the field, and Broader 
Impacts, which encompasses the potential to benefit society and contribute to the achievement 
of specific, desired societal outcomes. Within each of these two review criteria are five review 
elements (Figure 1). These five review elements line up remarkably well with the NIH core review 
criteria (Table 2), with both agencies’ criteria addressing a similar set of concepts but using distinct 
language to describe each criterion. 


• What is the potential for the proposed activity to (a) advance knowledge and 
understanding within its own field or across different fields (Intellectual Merit); and (b) 
benefit society or advance desired societal outcomes (Broader Impact)? 

• To what extent do the proposed activities suggest and explore creative, original, or 
potentially transformative concepts? 

• Is the plan for carrying out the proposed activities well-reasoned, well-organized, and 
based on a sound rationale? Does the plan incorporate a mechanism to assess success? 

• How well qualified is the individual, team, or institution to conduct the proposed 
activities? 

• Are there adequate resources available to the PI (either at the home institution or 
through collaborations) to carry out the proposed activities? 


Figure 1. NSF Merit Review Criteria (National Science Foundation, 2014) 
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Table 2. Comparison of the NIH and NSF research grant proposal review criteria 

Key Question 

NIH Core Review Criteria 3 

NSF Review Elements 15 



Intellectual Merit - Potential of 

Why does the 
research matter? 

Significance - project addresses an 
important problem or a critical 
barrier to progress in the field 

the activity to advance 
knowledge and understanding 

Broader Impact - Potential of the 
activity to benefit society 

How is the research 

new? 

Innovation - project challenges 
current paradigms by utilizing 
novel theoretical concepts, 

Creative, original, and 
transformative concepts and 

approaches or methodologies, 
instrumentation, or interventions 

activities 


How will the research 

be done? 

Approach - overall strategy, 
methodology, and analyses well- 
reasoned and appropriate to 
accomplish the specific aims of the 
project 

Well-reasoned, well-organized, 
rational plan for carrying out 
proposed activities and 
mechanism to assess success 


Environment - scientific 


In what context will 

environment in which the work 

Adequate resources available to 

the research be done? 

will be done contribute to the 
probability of success 

carry out the proposed activities 

What is special about 

Investigators - PD/PIs, 

Qualified individual, team, or 

the people doing the 

collaborators, and other researchers 

institution conducting the 

research? 

are well suited to the project 

proposed activities 


Overall Impact 0 - likelihood for the 

The potential to benefit society 

What is the return on 

project to exert a sustained, 

and contribute to the 

investment? 

powerful influence on the research 

achievement of specific, desired 


field(s) involved 

societal outcomes 


a See National Institutes of Health, 2015a; 2015b. 
b See National Science Foundation, 2014. 

c While Overall Impact is not considered as a core review criterion, it asks reviewers to take into 
consideration the five core review criteria as they assess the likelihood of the project to have a 
strong and sustained influence on the research field. 

NIH, National Institutes of Health; NSF, National Science Foundation; PD, program director; PI, 
principal investigator. 
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Table 3. Comparison of research grant proposal review criteria used by the NIH, NSF, and NEH 


Key Question 

NIH Core 

Criteria 1 ' 

NSF Merit Review Elements 15 

NEH Application 
Review Criteria" 



Intellectual Merit - Potential of 
the activity to advance knowledge 


Why does the 

Significance 

and understanding 

Humanities 

research matter? 

Broader Impact - Potential of the 
activity to benefit society 

Significance 



How is the 
research new? 

Innovation 

Creative, original, and 
transformative concepts and 
activities 

Quality of 

Innovation 



Well-reasoned, well-organized, 


How will the 

Approach 

rational plan for carrying out 

Project Feasibility 

research be done? 

proposed activities and 
mechanism to assess success 

and Work Plan 


In what context 

will the research 

be done? 

Environment 

Adequate resources available to 
carry out the proposed activities 

Project Feasibility 
and Work Plan 

What is special 
about the people 
doing the 
research? 

Investigators 

Qualified individual, team, or 
institution conducting the 
proposed activities 

Project Staff 
Qualifications 

What is the return 

on investment? 

Overall Impact 11 

The potential to benefit society 
and contribute to the achievement 
of specific, desired societal 

outcomes 

Overall Value to 

Humanities 

Scholarship 


a See National Institutes of Health, 2015a; 2015b. 

b See National Science Foundation, 2014. 

c See National Endowment for the Humanities, 2014; 2015. 

d While Overall Impact is not considered as a core review criterion, it asks reviewers to take into 
consideration the five core review criteria as they assess the likelihood of the project to have a 
strong and sustained influence on the research field. 

NIH, National Institutes of Health; NSF, National Science Foundation; NEH, National 
Endowment for the Humanities. 
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What about a non-science funding agency like the NEH ? While there is some variability between 
individual NEH grant programs, the NEH application review criteria are: Humanities Significance, 
Project Feasibility and Work Plan, Quality of Innovation, Project Staff Qualifications, and 
Overall Value to Humanities Scholarship (National Endowment for the Humanities, 2015a; 
2015b). The significance of the project includes its potential to enhance research, teaching, and 
learning in the humanities. The quality of innovation is evaluated in terms of the idea, approach, 
method, or digital technology (and the appropriateness of the technology) that will be used in 
the project. Reviewers also examine the qualifications, expertise, and levels of commitment of the 
project director and key project staff or contributors. The quality of the conception, definition, 
organization, and description of the project and the applicants clarity of expression, as well as 
the feasibility of the plan of work are also assessed. Finally, reviewers consider the likelihood that 
the project will stimulate or facilitate new research of value to scholars and general audiences in 
the humanities. Table 3 shows the NEH review criteria compared with those used by the NIH 
and NSF. Though there is not an exact match for the key question “In what context will the 
research be done?” (i.e., the research environment and available resources), this is evaluated in 
NEH proposals as part of the Project Feasibility and Work Plan. 

Comparing review criteria across federal agencies: Eight key questions 

In addition to the core review criteria mentioned above, funding agencies also typically ask 
reviewers to consider the project budget and the approach that will be used to evaluate project 
success. When we expanded the comparison of research grant proposal review criteria across 
10 US federal agencies, and included the budget and evaluation criteria, we revealed that all of 
the agencies’ review criteria aligned with a consistent set of eight key questions that reviewers 
consider when evaluating any type of research proposal (Table 4). 

The research grant proposal review criteria used by the 10 federal funding agencies are associated 
with these eight key questions (Table 5). We have already demonstrated that the question, “Why 
does it matter?”—which addresses the importance or significance of the proposed project— 
applies to similar review criteria from the NIH (Significance), NSF (Intellectual Merit), and 
the NEH (Humanities Significance) (National Endowment for the Humanities, 2015a; 2015b; 
National Institutes of Health, 2015a, 2015b; National Science Foundation, 2014). Likewise, 
ED evaluates the “Importance of the Problem” (US Department of Education, 2014); the DoD 
application review criteria includes “Importance” (Department of Defense, 2015); the VA and 
NASA each evaluate “Significance” (National Aeronautics and Space Administration, 2015; 
US Department of Veterans Affairs, 2015); the DOE looks at “Scientific and Technological 
Merit” (US Department of Energy, 2015); the USDA evaluates “Project Relevance” (United 
States Department of Agriculture, 2015); and the NEA assesses “Artistic Excellence” (National 
Endowment for the Arts, 2015). There are also parallels in the language used by each of the funders 
as they ask reviewers to assess proposed research project innovation or novelty, the approach or 
methodology to be used, the investigators or personnel involved, the environment and resources 
available, and the overall impact or value of the project (Table 5). 
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Table 4. Eight key questions considered by reviewers of research grant proposals and the 
associated review criteria terms used by 10 US federal funding agencies 


Key Question 

Review Criteria Terms 

Why does it matter? 

Significance 

Importance 

How is it new? 

Innovation 

Novelty 

Creativity 

How will it be done? 

Approach 

Plan 

Methodology 

Objectives 

Aims 

In what context will it be done? 

Environment 

Resources 

Populations 

Facilities 

What is special about the people involved? 

Investigators 

Organization 

People 

Researchers 

Personnel 

Partners 

Collaborators 

Staff 

What is the return on investment? 

Impact 

Value 

Relevance 

How effectively will the financial resources be 
managed? 

Budget 

How will success be determined? 

Evaluation 

Assessment 
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While all the agencies’ collective review criteria fall within the eight key questions, there is some 
variability across agencies. For example, the DOE does not have a clear review criterion for 
evaluating the overall impact or value of a project, equivalent to the key question “What is the 
return on investment?” Some agencies to do not explicitly include the budget as part of their 
review criteria, such as the NSF, VA, and USD A, while other agencies do not specifically ask 
for a plan to evaluate success of the project, including the NIFF, VA, DoD, DOE, USDA, or 
NEFF. Funders may also have unique review criteria. Unlike the other nine agencies evaluated, 
the DoD uses the review criterion “Application Presentation,” which assesses the writing, clarity, 
and presentation of the application components. Agencies may also have mission- or program- 
specific review criteria; for example, for certain applications, the NEA may evaluate the potential 
to reach underserved populations as part of “Artistic Merit.” Despite these differences, it is clear 
that for the 10 federal funding agencies examined, the review criteria used to evaluate research 
grant proposals are extraordinarily aligned. 

If we remember that all funding agencies are trying to evaluate research grant proposals to reach 
the same goals—to determine which projects fit within their mission and will provide a return on 
their financial investment—it is perhaps not all that surprising that the review criteria that federal 
funding agencies use are aligned. We further propose that funding announcements from any 
funder, including state agencies, local groups, and private philanthropic organizations, similarly 
ask for research grant proposals to answer some, if not all, of the eight key questions that emerged 
from our analysis of US federal funding agencies. Keeping these key questions in mind can help 
research administrators and research development offices, as well as proposal writers, decipher 
research grant proposal review criteria from almost any funding agency, thereby facilitating 
proposal development. 

For this article, we limited our analysis to the review criteria used across different US federal 
funders to evaluate research grant proposals, and did not include criteria used for other federal 
funding mechanisms, such as training grants or contract proposals. NIFF has compared the 
review criteria used across their various funding mechanisms, including research grants, grants 
for conferences and scientific meetings, small business innovation or technology transfer grants, 
fellowship and career development grants, and training grants, among others (National Institutes 
of FFealth, 2014). Again, while there are differences in the language used to describe each core 
review criterion across the various grant mechanisms, the concepts being reviewed—what is being 
done, why it is being done, how it is new, who is doing the work, and where it will be done—are 
essentially the same across each mechanism. 
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Conclusion 

We have demonstrated that research grant proposal review criteria are remarkably aligned across 
10 US federal funding agencies, despite the differences in their missions and the terminology each 
uses for its own review process (Table 5). Moreover, a set of only eight key questions summarizes the 
collective research grant proposal review criteria across all these federal agencies. While the sheer 
number of non-federal funding opportunities makes a similar comparative analysis of their review 
criteria impractical, we suggest that the eight key questions emerging from our analysis provide a 
starting point for researchers, research administrators, and funders to assess the review criteria used 
by most, if not all, other research funding opportunities. This is reasonable given that each funder 
is trying to achieve the same goal during the grant review process: find those research projects that 
fit the funder s mission and are worth its investment. Through this lens, the review criteria used for 
research proposals across agencies are easier to understand and address, which may encourage new 
investigators to apply for funding, and seasoned investigators and research development offices to 
consider a diversified set of funding sources for their research portfolios. Wfe also hope that this 
analysis provides guidance to other grantmakers as they develop review criteria for their own funding 
opportunities. For the 10 US federal agencies included here, we hope that the analysis serves as a 
starting point to develop even greater consistency across the review criteria—perhaps even a single 
canonic, cross-agency set of review criteria—used to evaluate federal research grant proposals. 
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